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Defense  Acquisitions:  Assessment  of  Institute  for  Defense  Analyses’  C-130  Avionics 
Modernization  Program  Analysis 

The  Air  Force’s  C-130  Avionics  Modernization  Program  (AMP),  which  entered  development  in 
2001 ,  was  to  standardize  and  upgrade  the  cockpit  and  avionics  for  various  configurations  of  the 
C-130  fleet.  The  C-130,  a  four-engine  turboprop  aircraft  also  known  as  “Hercules”  that  is  used 
primarily  for  military  transport,  was  originally  designed  in  the  1950s.  The  AMP  upgrades  were 
intended  to  ensure  that  the  C-130  can  satisfy  the  navigation  and  safety  requirements  it  needs  to 
operate  globally  while  at  the  same  time  reducing  crew  size  by  one.  The  program  would  also 
have  replaced  many  systems  for  which  manufacturers  no  longer  exist — a  situation  referred  to  as 
diminishing  manufacturing  sources. 

In  2007,  the  program  experienced  a  critical  breach  of  a  statutory  cost  threshold,1  and  in 
February  2012,  the  Department  of  Defense  (DOD)  proposed  cancelling  the  C-130  AMP.  The 
National  Defense  Authorization  Act  for  Fiscal  Year  2013  (NDAA)  stated  that  the  Secretary  of  the 
Air  Force  may  not  cancel  the  program  until  90  days  after  the  Secretary  provided  the 
Congressional  defense  committees  a  cost  benefit  analysis  conducted  by  the  Institute  for 
Defense  Analyses  (IDA).  The  NDAA  required  the  Secretary  to  seek  an  agreement  with  IDA  to 
conduct  an  independent  study  of  the  costs  and  benefits  of  upgrading  and  modernizing  the 
legacy  C-130H  airlift  fleet  via  AMP  compared  to  the  costs  and  benefits  of  a  reduced-scope 
program.2  IDA  began  its  work  in  March  2013  and  produced  the  required  study  in  September 
2013,  followed  by  a  detailed  annex  in  December  201 3. 3  The  National  Defense  Authorization  Act 
for  Fiscal  Year  2014  mandated  that  we  review  IDA’s  analysis.4  This  report  describes  IDA’s  study 
and  provides  our  assessment. 


1  DOD  is  required  to  notify  Congress  whenever  a  major  defense  acquisition  program’s  unit  cost  experiences  cost 
growth  that  exceeds  certain  thresholds.  This  is  commonly  referred  to  as  a  Nunn-McCurdy  breach.  Critical  breaches 
occur  when  the  program  acquisition  unit  cost  or  procurement  unit  cost  increases  by  at  least  25  percent  over  the 
current  baseline  estimate  or  at  least  50  percent  over  the  original.  When  a  program  experiences  a  Nunn-McCurdy 
breach  of  the  critical  cost  growth  threshold,  DOD  is  required  to  take  a  number  of  steps  including  reassessing  the 
program  and  submitting  a  certification  to  Congress  in  order  to  continue  the  program.  10  U.S.C.  §§  2433  and  2433a. 

2  Pub.  L.  No.  112-239,  §  143. 

3  Institute  for  Defense  Analyses,  C-130  Avionics  Modernization  Analysis,  (Alexandria,  VA:  Sept.  2013);  Institute  for 
Defense  Analyses,  C-130  Avionics  Modernization  Analysis:  Annex,  (Alexandria,  VA:  Dec.  2013). 

4  Pub.  L.  No.  113-66,  §  133  (2013).  The  NDAA  for  Fiscal  Year  2014  also  prohibited  the  Air  Force  from  using  any 
fiscal  year  2014  appropriations  to  cancel  or  modify  the  C-130  aircraft. 
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To  conduct  this  work,  we  obtained  IDA’s  September  2013  study  of  the  C-130  modernization 
alternatives  and  associated  December  2013  Annex  and  reviewed  the  data  and  assumptions  that 
IDA  used  to  reach  its  conclusions  and  recommendations.  GAO  focused  its  assessment  on  IDA’s 
cost  estimates  and  did  not  assess  the  operational  effectiveness  portion  of  IDA’s  study  as 
estimated  costs  were  the  major  discriminating  factor  in  IDA’s  conclusions.  We  compared  how 
IDA’s  estimates  were  developed  to  the  cost-estimating  best  practices  outlined  in  the  GAO  Cost 
Estimating  and  Assessment  Guide.5  Specifically,  we  analyzed  the  extent  to  which  IDA’s  cost 
estimates  met  the  characteristics  of  high-quality  cost  estimates.  We  also  discussed  the  study 
results  with  relevant  DOD  officials.  In  addition,  we  reviewed  IDA’s  C-130  AMP  annex  report  to 
determine  the  methodologies  used  to  support  its  findings.  More  detail  on  our  methodology  in 
applying  the  cost  estimating  best  practices  identified  in  the  GAO  Cost  Estimating  Guide  can  be 
found  in  enclosures  II  and  III. 

We  conducted  this  performance  audit  from  February  2014  through  May  2014  in  accordance  with 
generally  accepted  government  auditing  standards.  Those  standards  require  that  we  plan  and 
perform  our  audit  to  obtain  sufficient,  appropriate  evidence  to  provide  a  reasonable  basis  for  our 
findings  and  conclusions  based  on  our  audit  objectives.  We  believe  that  the  evidence  obtained 
provides  a  reasonable  basis  for  our  findings  and  conclusions  based  on  our  audit  objectives. 

Results  in  Brief 

IDA’s  study  is  comprised  of  two  key  components:  (1)  its  estimate  of  the  costs  of  AMP  and  two 
reduced  scope  alternatives  referred  to  as  Options  A  and  B,  which  offer  fewer  avionics  upgrades 
than  AMP  and,  unlike  AMP,  would  both  require  a  navigator  in  the  cockpit:  and  (2)  its 
assessment  of  the  operational  effectiveness — that  is,  benefits — of  AMP  and  each  alternative. 
IDA  estimated  that  that  AMP’s  costs  for  research,  development,  test  and  evaluation  (RDT&E), 
procurement,  and  operation  and  support  (O&S)  over  a  25-year  period  would  total  $2,137 
billion — significantly  higher  than  the  estimated  $1 .531  billion  for  Option  A  and  $530  million  for 
Option  B.  Regarding  benefits,  IDA  found  that  the  lower-cost  options  offered  nearly  as  much 
capability  as  AMP,  and  the  study  concluded  that  the  Air  Force  should  not  pursue  the  C-130 
AMP  program  as  currently  defined. 

We  found  that  IDA’s  cost  analysis  generally  followed  best  practices  for  high-quality  cost 
estimates,  which  are  critical  to  the  success  of  any  program.  We  analyzed  the  cost  estimating 
methodology  IDA  used  in  its  study  compared  to  best  practices  and  found  that  IDA’s  study 
substantially  met  two  of  the  characteristics  expected  of  high-quality  cost  estimates — that  they  be 
comprehensive  and  credible — and  partially  met  the  remaining  two  characteristics — being  well- 
documented  and  accurate. 

Background 

When  the  C-130  AMP  entered  development  in  2001 ,  it  was  intended  for  20  variants  of  the 
C-130  for  a  total  of  519  aircraft.  In  June  2007,  following  the  program’s  critical  Nunn-McCurdy 
breach,  DOD  conducted  a  five-month  review.  Subsequently,  in  accordance  with  10  U.S.C. 


5  GAO,  GAO  Cost  Estimating  and  Assessment  Guide:  Best  Practices  for  Developing  and  Managing  Capital  Program 
Costs,  GAO-09-3SP,  (Washington,  D.C.:  March  2009). 
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§  2433,  the  Under  Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics  certified  the 
C-130  AMP  program  to  Congress,  but  for  a  reduced  number  of  222  aircraft. 

The  C-130  AMP  entered  production  in  2010,  and  five  aircraft  were  produced  before  production 
was  halted.  Figure  1  shows  the  cockpit  of  a  C-130  with  AMP. 

Figure  1 :  Cockpit  of  a  C-1 30  with  AMP. 


Source:  C-130  AMP  Program  Office. 


in  the  President’s  Budget  for  Fiscal  Year  2013,  DOD  did  not  propose  any  funding  for  the  C-130 
AMP,  which  would  have  effectively  cancelled  the  program  in  favor  of  a  minimum 
Communication,  Navigation,  Surveillance/Air  Traffic  Management  (CNS/ATM)  solution  for  the 
legacy  C-130  combat  delivery  fleet.  According  to  DOD,  CNS/ATM  would  address  diminishing 
manufacturing  source  issues  and  complete  fleet  modifications  to  meet  required  navigation 
performance  mandated  by  2020.  Congress  did  not  authorize  any  modifications  to  the  program, 
but  rather  required  IDA’s  cost-benefit  study  of  the  issue. 

IDA  Study  Examined  Lower-Cost  Options  to  the  C-130  AMP 


IDA’s  cost-benefit  analysis  of  C-130  AMP  and  alternative  options  is  comprised  of  two  key 
components:  its  estimate  of  the  costs  of  AMP  and  the  alternatives,  and  its  assessment  of  the 
operational  effectiveness — that  is,  benefits — of  AMP  and  each  of  the  alternatives.  In  addition  to 
AMP,  IDA  examined  reduced-scope  alternatives  it  referred  to  as  Options  A  and  B.  It  reported 
that  AMP  was  the  most  sophisticated  of  the  three  alternatives  and  Option  B  the  least.  IDA’s 
study  noted  that  they  differ  in  many  details.  For  example,  the  alternatives  differ  with  respect  to 
level  of  area  navigation  and  required  navigation  performance,  potentially  imposing  future 
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restrictions  on  airfield  access  for  Options  A  and  B  and  aircraft  routing  for  Option  B.  IDA  also 
noted  that  the  most  significant  difference  is  the  removal  of  the  navigator  from  the  cockpit 
allowed  by  the  degree  of  automation,  a  head-up  display  and  advanced  flight  assistance 
equipment  by  the  AMP  alternative.6  Both  Options  A  and  B  would  have  fewer  avionics  upgrades 
than  AMP  and  would  require  the  navigator.  Unlike  Option  B,  Option  A  would  replace  older 
technology  gauges  with  multi-function  displays,  as  would  AMP.  In  addition,  unlike  AMP  and 
IDA’s  Option  A,  Option  B  would  not  replace  the  current  self-contained  navigation  system 
(SONS)  with  a  more  modern  SONS  capable  of  higher  location  and  planning  precision.  IDA 
concluded  that  the  lower-cost  options  to  C-130  AMP  had  nearly  as  much  capability  and  that  the 
Air  Force  should  not  pursue  the  C-130  AMP  program  as  currently  defined. 

To  provide  the  cost  portion  of  the  cost-benefit  analysis,  IDA  developed  25-year  cost  estimates 
for  the  alternatives  and  assessed  their  operational  implications.  IDA  developed  estimates  for 
RDT&E,  procurement,  and  O&S  for  the  alternatives  and  combined  those  estimates  to  determine 
overall  25-year  costs  for  each  alternative.  IDA  assumed  a  fleet  of  192  C-130Hs  for  all  options, 
the  number  of  aircraft  defined  in  the  Air  Force  Capability  Development  Document;  appropriately 
excluded  the  approximately  $1 .7  billion  previously  spent  on  the  AMP  program;  and  estimated 
the  costs  in  constant  fiscal  year  2013  dollars.  The  results  of  IDA’s  analysis  of  the  costs  of  AMP 
and  Options  A  and  B  are  presented  in  table  1. 


Table  1:  IDA  Cost  Estimates  for  Three  C-130  Modernization  Alternatives 

C-130  AMP 

Option  A 

Option  B 

Cost  Estimates  (constant  fiscal  year  2013  dollars  in  millions) 

RDT&E 

$104 

$437 

$140 

Procurement 

3,060 

1,320 

480 

O&S  Savings3 

-1,027 

-226 

-90 

Totals 

2,137 

1,531 

530 

Source:  GAO  presentation  of  IDA  results 

aThese  estimates  represent  solely  the  avionics-related  O&S  costs  and  savings  to  be  incurred  as  a  result  of 
implementing  each  of  the  alternatives.  A  negative  amount  depicts  a  net  savings  for  that  option. 


IDA’s  estimates  showed  that  that  the  AMP’s  costs  for  RDT&E,  procurement,  and  O&S  over  a 
25-year  period  were  significantly  higher  than  the  alternative  options  it  considered.  Specifically, 
the  analysis  shows  while  Options  A  and  B  have  higher  development  costs  and  lower  O&S 
savings  (due  primarily  to  the  fact  that  they  do  not  enable  the  reduction  of  one  crew  member), 
these  costs  are  more  than  offset  by  lower  procurement  costs. 

Regarding  overall  effectiveness,  IDA  found  the  AMP’s  operational  effectiveness  advantage  over 
Options  A  and  B  to  be  marginal.  Specifically,  the  study  found  Option  A  to  be  at  least  as 
operationally  effective  as  AMP  since  both  AMP  and  Option  A  address  the  diminishing 
manufacturing  sources  problem.  (See  enclosure  I  for  further  details  on  IDA’s  analyses  and 
results.) 

IDA’s  Cost  Estimating  Generally  Followed  Best  Practices 

Given  that  IDA  reported  finding  Option  A  at  least  as  operationally  effective  as  AMP,  and  that 
estimated  costs  were  the  apparent  discriminating  factor  in  IDA’s  recommendation  that  the  Air 
Force  not  pursue  AMP,  we  limited  our  assessment  of  IDA’s  cost  estimating  quality  to  AMP  and 


6  These  capabilities  are  similar  to  those  found  in  the  C-130J,  which  also  operates  without  a  navigator 
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Option  A.  We  found  that  IDA’s  cost  analysis  either  substantially  or  partially  met  each  of  the  four 
characteristics  expected  of  high  quality  cost  estimates.  Such  estimates  are  critical  to  the 
success  of  any  program  as  they  provide  the  basis  for  informed  investment  decision  making, 
realistic  budget  formulation  and  program  resourcing,  meaningful  progress  measurement, 
proactive  course  correction  when  warranted,  and  accountability  for  results.  These 
characteristics  are  described  briefly  below  in  table  2.  A  more  detailed  description  of  each 
characteristic  of  high-quality  cost  estimates  and  our  complete  assessment  can  be  found  in 
enclosure  II. 

Specifically,  we  analyzed  the  cost  estimating  practices  used  by  IDA  in  its  analysis  of  the  C-130 
AMP  and  alternative  options  against  best  practices  for  high-quality  cost  estimates.  As  table  2 
shows,  we  found  that  the  cost  estimates  in  IDA’s  study  substantially  met  two  of  the 
characteristics  expected  of  high-quality  cost  estimates — comprehensiveness  and  credibility — 
and  partially  meet  the  remaining  two  characteristics — being  well-documented  and  accurate. 


Table  2:  Evaluation  of  IDA’s  Cost  Estimates 


Characteristics  of  high-quality  cost  estimates 

Our  evaluation3 

1.  Comprehensive  (estimate  is  complete  and  accounts  for  all  possible  costs) 

Substantially  Met 

2.  Well  documented  (detail  and  quality  of  the  information  used) 

Partially  Met 

3.  Accurate  (should  be  unbiased  and  neither  overly  conservative  or  optimistic) 

Partially  Met 

4.  Credible  (to  include  sensitivity  analysis,  risk  and  uncertainty  analysis) 

Substantially  Met 

Source:  GAO  assessment. 

a  For  the  purposes  of  our  analysis,  we  determined  that  a  characteristic  was  “met”  if  IDA  provided  complete  data  that 
satisfies  the  entire  criterion;  “substantially  met”  if  IDA  provided  data  that  satisfies  a  large  portion  of  the  criterion; 
“partially  met”  if  IDA  provided  evidence  that  satisfies  about  half  of  the  criterion;  “minimally  met”  if  IDA  provided 
evidence  that  satisfies  a  small  portion  of  the  criterion;  and  “not  met”  If  IDA  provided  no  data  that  satisfies  any  of  the 
criterion. 

More  specifically  we  found  the  following: 

•  Comprehensiveness:  We  found  the  estimates  to  be  substantially  comprehensive 
because  the  estimates  appeared  to  include  the  pertinent  life  cycle  costs  and  the 
estimates’  cost  element  structure  was  at  an  sufficient  level  of  detail  considering  the 
purpose  of  the  study  was  a  cost  benefit  analysis  comparison  of  alternatives.  However, 
we  also  found  that  few  of  the  typical  technical  baseline  elements  were  described  in  the 
study  and  that  ground  rules  and  assumptions  were  incomplete.  Specifically,  the  study 
itself  did  not  identify  inflation  indices  or  their  source,  although  IDA  subsequently 
identified  that  they  used  the  OSD  Comptroller’s  Green  Book  for  indices. 

•  Documentation:  We  found  the  estimates  partially  met  best  practices  for  this 
characteristic.  The  study  documentation  described  the  estimating  methodology  used  to 
derive  each  cost  element,  but  did  not  describe  in  detail  the  calculations  performed  to 
arrive  at  the  estimate.  Additionally,  the  documentation  described  step-by-step  how  the 
estimate  was  developed,  but  it  did  not  contain  sufficiently  detailed  data  to  allow  a  cost 
analyst  unfamiliar  with  the  program  to  replicate  the  estimate.  Furthermore,  the 
documentation  does  not  contain  the  source  data  used  to  develop  the  estimates. 

•  Accuracy:  We  found  the  estimates  partially  met  best  practices  for  this  characteristic. 
The  estimates  were  based  on  actual  costs  from  the  C-130  AMP  and  other  programs, 
program  office  estimates  for  C-130  AMP  and  other  similar  programs,  cost  estimating 
relationships,  and  factors,  but  because  the  historical  source  data  were  not  included,  we 
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could  not  assess  the  reliability  and  relevance  of  the  data.  Additionally,  while  the  study 
team  performed  uncertainty  analysis  and  displayed  the  point  estimates  and  low  and  high 
ranges  of  the  estimate,  they  did  not  show  the  confidence  levels  of  the  point  estimates  to 
allow  us  to  determine  if  the  estimates  were  unbiased. 

•  Credibility:  We  found  the  estimates  substantially  met  best  practices  for  this 

characteristic.  The  study  included  a  risk  and  uncertainty  analysis  and  displayed  the 
results  in  the  difference  between  the  net  present  value  of  AMP  and  Option  A.  However, 
the  documentation  did  not  contain  cumulative  probability  distributions,  known  as  “S 
curves,”  of  the  alternative  estimates.  Further,  the  study  sensitivity  analysis  was  limited  to 
examination  of  the  most  significant  contributors  to  AMP  cost  variance. 

Life  cycle  cost  analyses  may  be  sensitive  to  underlying  assumptions,  such  as,  the  time  period 
covered  by  an  analysis.  In  this  case,  IDA  was  tasked  with  and  developed  costs  for  a  25-year 
period.  Over  that  period,  it  estimated  that  the  estimate  for  Option  A  was  nearly  $600  million  less 
than  for  AMP.  Comparatively,  in  the  last  year  of  its  study,  IDA  estimated  AMP  annual  savings 
that  were  $47  million  greater  than  for  Option  A.  Hence,  had  the  time  period  been  less  than  25 
years,  the  difference  between  the  AMP  and  Option  A  estimates  could  have  been  greater;  had 
the  time  period  been  longer  than  25  years,  the  difference  could  have  been  less.  Other 
assumptions  can  similarly  influence  results.  For  example,  IDA  noted  in  its  conclusions  that 
factors  outside  of  the  scope  of  its  study  — including  retirement  schedules  for  C-130s  and  the 
acquisition  of  new  C-130Js — should  inform  decisions  determining  the  specific  upgrade  program 
chosen. 

We  are  not  making  recommendations  in  this  report. 

Agency  Comments  and  Our  Evaluation 

DOD  and  IDA  provided  comments  on  a  draft  of  this  report,  which  are  reprinted  in  enclosures  IV 
and  V,  respectively.  In  its  comments,  DOD  referred  to  our  work  as  validating  IDA’s  analysis.  We 
note  that  we  analyzed  the  cost  estimating  methodology  IDA  used  in  its  study  compared  to  best 
practices  for  producing  reliable  and  valid  cost  estimates.  However,  we  did  not  produce  separate 
estimates  nor  did  we  evaluate  IDA’s  recommendations.  DOD  also  commented  that  it  took  into 
consideration  the  data  sources  that  supported  IDA’s  conclusions  when  reviewing  IDA’s 
recommendations. 

In  addition,  IDA  commented  that  it  agreed  with  our  assessment  that  its  cost  analysis  generally 
followed  best  practices  for  high  quality  estimates.  Specifically,  IDA  agreed  with  our  assessment 
that  its  cost  analysis  was  comprehensive  and  credible,  but  believed  they  warranted  a  higher 
rating  of  “met”,  rather  than  “substantially  met”  for  these  two  characteristics.  IDA  also  noted  that  it 
strongly  believes  our  assessments  of  its  study  against  the  remaining  two  characteristics — being 
well-documented  and  accurate — should  have  been  higher.  As  noted  in  our  report,  we  assessed 
IDA’s  cost  estimating  as  having  partially  met  the  criteria  for  being  well  documented  because  it 
did  not,  in  our  opinion,  contain  sufficiently  detailed  data  to  allow  a  cost  analyst  unfamiliar  with 
the  program  to  replicate  the  estimate.  Similarly,  we  assessed  IDA’s  estimate  as  having  only 
partially  met  the  best  practice  of  accuracy  because  the  historical  source  data  were  not  included, 
and  we  could  not,  therefore,  assess  the  reliability  and  relevance  of  the  data.  Our  assessment 
was  governed  by  the  information  contained  in  the  report  itself,  which  was  limited  in  these  two 
areas.  That  does  not  necessarily  mean  that  the  information  did  not  exist  elsewhere.  IDA  also 
commented  that  we  did  not  find  any  evidence  that  its  cost  estimates  were  not  accurate  and  that 
nothing  in  our  assessment  undermines  its  findings  and  recommendations. 
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We  are  sending  copies  of  this  report  to  interested  congressional  committees;  the  Secretary  of 
Defense;  the  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics;  and  the 
Secretary  of  the  Air  Force.  This  report  also  is  available  at  no  charge  on  GAO’s  website  at 
http://www.gao.gov. 

Should  you  or  your  staff  have  any  questions  on  the  matters  covered  in  this  report,  please 
contact  me  at  (202)  512-4841  or  sullivanm@gao.gov.  Contact  points  for  our  Offices  of 
Congressional  Relations  and  Public  Affairs  may  be  found  on  the  last  page  of  this  report.  Key 
contributors  to  this  report  were  Bruce  H.  Thomas,  Assistant  Director;  Marvin  Bonner,  Analyst-in- 
Charge;  John  Beauchamp;  Susan  Ditto;  Karen  Richey;  Brian  Bothwell;  Marie  Ahearn;  and 
Kenneth  E.  Patton. 


Michael  J.  Sullivan 

Director,  Acquisition  and  Sourcing  Management 
Enclosures  -  5 
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List  of  Committees 


The  Honorable  Carl  Levin 
Chairman 

The  Honorable  James  Inhofe 
Ranking  Member 
Committee  on  Armed  Services 
United  States  Senate 

The  Honorable  Richard  J.  Durbin 
Chairman 

The  Honorable  Thad  Cochran 
Ranking  Member 
Subcommittee  on  Defense 
Committee  on  Appropriations 
United  States  Senate 

The  Honorable  Howard  P.  “Buck”  McKeon 
Chairman 

The  Honorable  Adam  Smith 
Ranking  Member 
Committee  on  Armed  Services 
House  of  Representatives 

The  Honorable  Rodney  Frelinghuysen 
Chairman 

The  Honorable  Pete  Visclosky 
Ranking  Member 
Subcommittee  on  Defense 
Committee  on  Appropriations 
House  of  Representatives 
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Enclosure  I:  IDA’s  Cost  Estimating  and  Operational  Effectiveness  Methodology  and 
Results 

RDT&E  Cost  Estimates 


In  developing  estimated  RDT&E  costs  associated  with  AMP  and  each  of  the  options,  IDA 
reported  that  it  included  nonrecurring  design  costs  as  well  as  the  cost  of  building  the  test  article 
and  conducting  associated  testing.  Utilizing  data  from  the  last  C-130  AMP  Selected  Acquisition 
Report,  the  C-130J  Avionics  upgrade,  and  the  KC-10  AMP,  IDA  estimated  that  the  remaining 
development  costs  for  AMP  to  be  about  $104  million.  To  estimate  the  development  costs  for 
Options  A  and  B,  IDA  utilized  data  from  the  C-130  AMP  program,  removing  costs  associated 
with  reduced  content  (eliminated  requirements  and  fewer  variants  and  conversions)  and  sunk 
costs.  IDA  then  accounted  for  the  reduced  content  of  Options  A  and  B,  determining  the  subset 
of  equipment  and  associated  cost  necessary  for  the  functionality  of  Options  A  and  B,  including 
government  costs,  as  it  did  with  AMP.  Table  3  summarizes  IDA’s  RDT&E  cost  estimates  for 
C-130  AMP  and  the  alternatives  considered. 


Table  3:  IDA’s  C-130  AMP  and  Options  RDT&E  Cost  Estimates 

C-130  AMP 

Option  A 

Option  B 

Cost  Estimates  (constant  fiscal  year  2013  dollars  in  millions) 

IDA’s  estimates 

$104 

$437 

$140 

Source:  GAO  presentation  of  IDA  results 


As  a  cross-check  on  its  RDT&E  cost  estimates  for  Options  A  and  B,  IDA  examined  additional 
data.  They  collected  data  from  the  C-130J  avionics  upgrade  and  reduced  its  content  to  derive  a 
comparison  cost  estimate.  IDA  also  developed  comparison  estimates  based  on  the  C-5  AMP 
development  program  (deemed  similar  to  Option  A)  and  the  KC-10  AMP  development  program 
(deemed  similar  to  Option  B).  In  addition,  IDA  reported  that  it  examined  Air  Force  Cost  Analysis 
Agency  (AFCAA)  estimates  for  the  two  successor  programs  the  Air  Force  had  proposed  for 
C-130  AMP — sometimes  called  the  “Optimize”  and  the  “Minimize”  CNS/ATM  programs.  Table  4 
provides  IDA’s  Option  A  and  Option  B  RDT&E  cost  estimate  and  the  associated  cross-check 
estimates. 


Table  4:  IDA’s  Cost  and  Cross-check  Estimates  for  RDT&E  for  Options  A  and  B 

Option  A 

Option  B 

Cost  Estimates  (constant  fiscal  year  2013  dollars  in  millions) 

IDA  cost  estimate 

$437 

$140 

IDA  Adjusted  C-130J  avionics  upgrade  analogy  cross-check 

418 

220 

IDA  C-5  AMP  analogy  cross-check 

501 

IDA  KC-10  AMP  analogy  cross-check 

57 

AFCAA  “Maximize”  and  “Minimize”  estimates  cross-check 

368 

185 

Source:  GAO  presentation  of  IDA  results 


As  the  table  above  shows,  the  IDA  estimate  for  RDT&E  costs  fall  between  the  upper  and  lower 
bounds  of  the  other  estimates. 
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Procurement  Cost  Estimates 


In  discussing  its  estimation  of  procurement  costs,  IDA  noted  that  programs  that  modify  existing 
platforms  have  distinctive  features  that  result  in  cost  elements  that  are  unique  relative  to  a  new¬ 
start  acquisition.  In  developing  its  procurement  cost  estimates  for  the  C-130  AMP  and  the 
alternatives,  IDA  examined  those  elements:  kits  (i.e.  hardware),  installation,  and  other 
acquisition  costs  (e.g.  training  equipment,  peculiar  support  equipment,  depot  standup,  initial 
spares,  and  interim  contractor  support),  and  program  support  (e.g.  system  engineering  and 
program  management).  IDA  reported  that  it  utilized  acquisition  schedules  based  on  plans  and 
estimates  previously  put  forth  by  the  Air  Force  and  assumed  a  sole-source  procurement 
strategy  for  each  of  the  three  alternatives. 

IDA  noted  that  estimating  procurement  costs  for  AMP  was  unique  as  cost  data  for  completed 
production-configuration  kits  and  installations  for  five  aircraft  were  available.  It  estimated 
procurement  costs  for  the  AMP  by  defining  price-improvement  curves  for  the  kits  and  installation 
and  extrapolating  previous  experience  forward  to  define  fly-away  cost  (the  sum  of  kits  and 
installation  costs).  IDA  reported  that  it  estimated  other  acquisition  costs  using  the  relationships 
of  other  costs  to  flyaway  costs  suggested  by  historical  data  for  C-130  and  other  transport  aircraft 
modification  programs.  For  Options  A  and  B,  IDA’s  estimates  used  “catalog”  prices  presented 
by  AFCAA  for  off-the-shelf  components  that  would  meet  Option  A  and  B  defined  capabilities. 

IDA  reported  that  it  estimated  the  cost  of  associated  hardware  needed  for  the  installation  of 
those  off-the-shelf  components  (e.g.  brackets,  mounts,  cables,  etc.)  and  the  cost  of  installation 
itself  based  on  a  combination  of  historical  factors  and  other  analyses.  IDA  also  calculated 
average  procurement  unit  costs  for  each  of  the  alternatives.  Table  5  summarizes  IDA’s 
procurement  total  and  average  unit  (per  aircraft)  cost  estimates  for  C-130  AMP  and  the 
alternatives  considered. 


Table  5:  IDA’s  C-130  AMP  and  Options  Procurement  Cost  Estimates 

C-130  AMP 

Option  A 

Option  B 

Cost  Estimates  (constant  fiscal  year  2013  dollars  in  millions) 

Total  estimated  procurement  costs 

$3,060 

$1,320 

$480 

Average  estimated  unit  procurement  cost 

16.4 

7.0 

2.5 

Source:  GAO  presentation  of  IDA  results 


IDA  calculated  the  average  procurement  unit  costs  to  enable  comparison  with  estimates  it 
developed  based  on  information  from  other  sources,  thereby  providing  a  cross-check  against  its 
estimates  for  AMP  and  Option  A.7  For  example,  IDA  developed  two  cross-check  estimates  for 
AMP:  one  based  on  the  2010  Selected  Acquisition  Report  produced  for  the  C-130  AMP 
program;  the  other  based  on  a  C-130  AMP  System  Program  Office  “should  cost”  estimate.8  To 
make  these  estimates  comparable,  IDA  adjusted  the  unit  costs  from  them  to  account  for 
differences  in  their  associated  quantities  with  the  number  of  AMP  aircraft  used  in  calculating  its 
estimate  and  adjusted  them  to  constant  fiscal  year  2013  dollars.  Table  6  provides  IDA’s  AMP 
procurement  cost  and  associated  cross-check  estimates. 


7  The  IDA  study  did  not  provide  procurement  cross-check  estimate  calculations  for  Option  B  or  explain  why  it  did  not. 

8  In  accordance  with  direction  provided  in  Interim  DOD  Instruction  5000.2.  “Operation  of  the  Defense  Acquisition 
System,”  and  memorandums  issued  by  the  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics, 
each  program  is  to  conduct  a  “should-cost”  analysis  which  includes  justifying  each  cost  under  the  program’s  control 
with  the  aim  of  reducing  negotiated  prices  for  contracts  and  obtaining  other  efficiencies  in  program  execution. 
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Table  6:  IDA’s  Cost  and  Cross-check  Estimates  for  Procurement  of  AMP 

C-130  AMP 

Cost  Estimates  (constant  fiscal  year  2013  dollars  in  millions) 

IDA  estimated  average  unit  procurement  cost 

$16.4 

IDA  cross-check  estimate  based  on  C-130  AMP  Final  SAR 

19.5 

IDA  cross-check  estimate  based  on  adjusted  “should  cost” 

14.3 

Source:  GAO  presentation  of  IDA  results 


As  can  be  seen,  the  IDA  estimate  for  AMP  procurement  costs  falls  between  the  upper  and  lower 
bounds  of  the  other  estimates. 

O&S  Cost  Estimates 


IDA  estimated  O&S  costs  for  the  three  alternatives — O&S  for  192  aircraft  (including  aircraft 
previously  completed)  over  a  25  year  period — in  constant  fiscal  year  2013  dollars  and  also 
calculated  a  net  present  value  for  each.  IDA  reported  that  it  included  only  the  incremental  O&S 
costs  associated  with  each  alternative,  which  for  all  three  modernization  alternatives  amounted 
to  savings  from  the  current  C-130  fleet.  Elements  of  O&S  costs  included  personnel,  indirect 
costs,  training,  fuel,  depot-level  reparables,  diminishing  manufacturing  sources,  consumables, 
sustaining  engineering,  program  management,  support  equipment,  and  software. 

IDA’s  study  indicates  that  the  largest  driver  of  cost  savings  in  its  O&S  estimate  for  AMP  relates 
to  the  elimination  of  the  navigator  position  from  the  C-130  cockpit — savings  from  personnel 
costs  as  well  as  training.  To  calculate  these  savings,  IDA  reported  that  it  adjusted  information 
from  a  2010  AMP  manpower  estimate  report  as  the  basis  for  manpower  reductions  and  applied 
current  salary,  training  costs  and  turnover  rate  information  from  the  Air  Force.  It  assumed  no 
navigator-related  savings  for  Options  A  and  B.  Another  large  driver  of  AMP  cost  savings  relative 
to  the  other  options  was  related  to  fuel  consumption.  IDA  estimated  the  incremental  O&S  costs 
for  fuel  by  factoring  weight  considerations  for  the  avionic  systems  added  or  removed  by  each 
alternative  along  with  costs  relating  to  potential  noncompliance  with  relevant  future 
requirements.  IDA  noted  that  while  AMP  allows  unrestricted  global  operations  by  meeting 
required  navigational  performance  and  area  navigation,  aircraft  that  do  not  meet  those 
requirements — such  as  Options  A  and  B — can  have  restrictions  on  maximum  altitude  and 
landings,  which  can  reduce  flying  efficiency.  To  account  for  this,  IDA  studied  historical  C-130H 
flying  and  estimated  the  potential  cost  associated  with  not  meeting  future  CNS/ATM 
requirements.  In  other  O&S  cost  aspects,  IDA  estimated  Options  A  and  B  to  be  less  costly  than 
the  AMP.  Table  7  provides  IDA’s  total  calculated  25-year  life  cycle  O&S  costs  (actually  savings) 
for  each  of  the  alternatives. 


Table  7:  IDA’s  C-130  AMP  and  Options  O&S  Savings  Estimates 

C-130  AMP 

Option  A 

Option  B 

Cost  Estimates  (constant  fiscal  year  2013  dollars  in  millions) 

IDA’s  total  25-year  O&S  estimates 

$-1,027 

$-226 

$-90 

Source:  GAO  presentation  of  IDA  results 
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Total  Estimated  Costs 


Combining  its  RDT&E,  procurement,  and  O&S  cost  estimates  from  the  options,  IDA  calculated 
total  costs  for  each  of  the  options  in  fiscal  year  2013  constant  dollars.  In  addition,  it  discounted 
those  amounts  to  compute  net  present  values  for  each  of  the  options.  Table  8  provides  IDA’s 
calculated  total  costs  and  net  present  values  for  the  options. 


Table  8:  IDA’s  25-year  Total  Cost  (RDT&E,  Procurement,  and  O&S)  Estimates  for  C-130  AMP  and  Options 

C-130  AMP 

Option  A  Option  B 

IDA’s  25-year  total  estimates  (constant  fiscal  year  2013  dollars  in  billions) 

$2.12 

$1.53  $.53 

IDA’s  25-year  net  present  value  estimates  (in  billions  of  discounted  dollars) 

2.11 

1.46  .51 

Source:  GAO  presentation  of  IDA  results 


IDA  Took  An  Additional  Step  to  Address  Estimation  Uncertainty 

To  deal  with  the  cost  estimating  uncertainty,  IDA  performed  cost-risk  analysis  developing  a  cost- 
risk  model  and  utilizing  Monte  Carlo  simulation.  For  all  but  two  of  the  variables  in  its  model,  IDA 
utilized  triangular  distributions  defined  by  a  most  likely  value — its  point  estimates — and  lower 
and  upper  limits.  For  example,  the  RDT&E  upper  and  lower  limits  were  based  on  the  range  of 
RDT&E  costs  experienced  on  similar  programs.  For  cost  risk  relating  to  the  elimination  of  a 
navigator  position  from  AMP,  based  on  discussions  with  Air  Force  and  U.S.  Coast  Guard 
Personnel,  IDA  considered  the  case  where  some  tactical  missions  (e.g.  airdrop)  require  a 
navigator  or  sensor  operator  even  with  AMP  aircraft.  To  account  for  this  risk,  using  analytical 
judgment,  IDA  defined  a  discrete  distribution.  Lastly,  IDA  accounted  for  variations  in  flying  hours 
by  utilizing  a  normal  distribution  centered  on  the  C-130H  planned  flying  program.  IDA’s  analysis 
accounted  for  correlation  among  inputs.  For  example,  IDA’s  model  included  positive  correlation 
between  procurement  and  depot-level  costs  as  both  depend  on  the  cost  of  materials. 

IDA  ran  one  hundred  thousand  simulations,  finding  that  in  98  percent  of  the  simulations  AMP’s 
net  present  value  cost  was  greater  than  Option  A.  IDA  also  performed  sensitivity  analysis  to 
identify  facts  that  have  the  most  effect  on  the  AMP  estimates,  finding  that  AMP  procurement 
cost  was  the  most  significant  contributor  to  cost  estimating  variance,  accounting  of  72  percent  of 
the  observed  variance.  The  second  largest  contributor  was  AMP  depot-level  reparable  cost, 
which  accounted  for  13  percent. 

IDA’s  Analysis  of  Operational  Effectiveness 

IDA  defined  effectiveness  as  the  ability  of  a  modernized  C-130  fleet  to  enable  C-130 
operations — consisting  of  combat  delivery,  non-combat  cargo  and  passenger  movement,  and 
the  necessary  functions  (e.g.  training)  to  support  the  first  two.  IDA  noted  that  given  that  aircraft 
availability  is  integrally  tied  to  all  aspects  of  operations,  it  should  be  included  as  a  component  of 
effectiveness.  Overall,  IDA  found  the  AMP’s  operational  effectiveness  advantage  over  Options 
A  and  B  to  be  marginal.  It  stated  that  it  found  Option  A  to  be  at  least  as  operationally  effective 
as  AMP  since  both  AMP  and  Option  A  address  diminishing  manufacturing  sources.  Additionally, 
it  noted  that  training  and  scheduling  inefficiencies  have  largely  been  solved  outside  of  avionics 
modernization  and  crew  work  loads  are  not  an  issue  with  Option  A  as  it  retains  the  navigator 
position. 

In  assessing  the  aircraft  availability  of  the  AMP  and  alternatives,  IDA  estimated  the  upper  bound 
on  expected  improvements  to  the  mission  capable  rate  as  a  measure  of  availability  as  the 
mission  capable  rate  is  an  indicator  of  whether  the  C-130  fleet  is  ready  to  perform  its  mission 
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when  needed.9  IDA  examined  historical  mission  readiness  data  for  the  C-130H  fleet  in 
determining  the  upper  bound.  It  found  that  the  best-case  possible  mission  capable  rate 
improvement  that  AMP  could  achieve  was  1 .4  percent.  IDA  then  looked  at  the  other  extreme 
Option  B.  IDA  found  a  best-case  improvement  for  Option  B  of  only  0.5  percent.  IDA  concluded 
that  mission  capable  rate  improvement  is  not  a  discriminating  factor  with  regard  to  operational 
effectiveness. 

Another  area  included  in  IDA’s  operational  effectiveness  assessment  was  combat  delivery  and 
passenger  and  cargo  movement.  IDA  chose  to  include  these  areas  as  part  of  the  effectiveness 
assessment  because  the  primary  mission  of  the  C-130  is  to  deliver  troops  and  cargo  to  the 
forward  operating  locations  and  drop-zones  in  a  combat  environment.  Although  the  AMP  and 
the  reduced-scope  alternatives  were  not  intended  to  change  combat  delivery  capabilities, 
elimination  of  the  navigator  in  the  AMP  redistributes  the  workload  and  responsibilities  of  the 
remaining  crew  members  during  combat  delivery.  In  evaluating  the  effectiveness  of  combat 
delivery  of  AMP  and  the  alternatives,  IDA  examined  whether  AMP’s  combat  delivery  would  be 
affected  by  replacing  the  navigator  position.  IDA  found  that  some  subject  matter  experts  believe 
that  removal  of  the  navigator  position  would  increase  crew  workload,  especially  in  high-threat 
environments.  IDA  reported  that  Operations  Iraqi  Freedom  and  Enduring  Freedom,  with  the  C- 
130J  and  C-17  aircraft,  which  also  eliminate  the  navigator  position,  indicates  that  some 
percentage  of  airlift  missions  require  augmentation  with  a  third  pilot  to  mitigate  crew  workload. 

In  summary,  IDA  noted  that  if  additional  crew  members  are  needed  for  AMP,  then  AMP  will  cost 
more  than  it  estimated  in  calculating  AMP  O&S  costs  as  it  gave  AMP  full  credit  for  removal  of 
the  navigator  in  determining  its  base  AMP  O&S  costs. 

In  evaluating  non-combat  cargo  and  passenger  movement,  IDA  sought  to  determine  if 
differences  in  required  area  navigation  and  navigation  performance  for  AMP  and  the  alternative 
options  might  impose  restrictions  on  aircraft  routing  and  airfield  access,  thereby  affecting 
operational  effectiveness.  To  determine  the  effectiveness  of  the  modernization  alternatives  in 
this  respect,  IDA  used  two  methodologies — assessing  the  effect  of  limited  required  area 
navigation  and  navigation  performance  capabilities  on  airfield  approaches  and  assessing  the 
impact  to  flying  high-altitude  routes.  IDA  found  that  AMP  provides  the  most  operational  flexibility 
if  the  aviation  community  is  converted  to  a  predominately  precision  navigation-based  system 
that  completely  replaces  the  current  ground-based  system.  There  are  currently  no  definitive 
FAA  plans  to  do  so.  IDA  also  found  that  the  AMP  alternatives  could  experience  delays  at  large 
airfields  because  of  their  limited  navigation  approach  capability — potentially  affecting  up  to  33 
percent  of  C-1 30H  sorties  if  certain  larger  airfields  are  required  to  adopt  the  newer  technology. 

In  assessing  the  effects  to  flying  high  altitude  routes,  IDA  concluded  that  access  to  higher 
altitudes  afforded  by  the  AMP  and  Option  A  navigation  capability  result  in  modest  fuel  savings 
and  increased  payload  capacity  at  extreme  flying  ranges  and  payloads,  but  may  not  be 
operationally  significant  for  the  most  common  routes  and  cargo  weights. 

IDA  also  assessed  operations  support — involving  primarily  improvements  to  training  and 
scheduling — for  AMP  and  the  alternatives.  IDA’s  objective  was  to  determine  if  the  different 
modernization  options  sufficiently  standardized  the  C-130H  fleet  to  eliminate  or  reduce 
inefficiencies  associated  with  tailoring  training  to  each  variant  of  the  aircraft.  Standardization 
had  been  cited  as  a  key  reason  for  the  C-130  AMP.  To  address  this  question,  IDA  obtained 
equipment  lists  for  the  C130H  and  sought  to  determine  which  avionics  equipment  varied  across 
the  C-130  fleet.  IDA  determined  that  there  are  eight  avionics  components  that  are  the  primary 


9  Mission  capable  rate  is  a  measure  of  an  aircraft’s  readiness  to  perform  its  missions. 
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drivers  for  variations  in  C-130H  training.  Option  A  would  standardize  four  of  those  differences 
and  Option  B  one,  while  AMP  would  eliminate  all  of  the  avionics  component  differences.  IDA 
noted  that  while  differences  in  training  would  exist  under  Options  A  and  B,  the  present  day 
version  of  this  training  is  no  longer  the  concern  it  was  a  decade  ago  as  a  result  of  the  selective 
retirement  of  aircraft  that  has  minimized  the  number  of  C-130  variants.  IDA  also  noted  that  the 
fewer  number  of  C-130H  variants  and  sub-variants  facilitates  mission  planning  and  scheduling. 
In  summary,  IDA  found  that  while  neither  Options  A  nor  B  completely  eliminates  the  need  for 
differences  in  training,  the  significantly  reduced  number  of  C-130  variants  has  largely  solved  the 
training  problems  to  the  point  that  they  are  no  longer  of  operational  concern. 
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Enclosure  II:  Characteristics  of  High-Quality  and  Reliable  Cost  Estimates  and  Our 
Detailed  Assessment  of  IDA’s  Cost-Benefit  Analysis 

Our  research  has  identified  a  number  of  best  practices  that  are  the  basis  of  effective  program 
cost  estimating  and  should  result  in  reliable  and  valid  cost  estimates  that  management  can  use 
for  making  informed  decisions.  These  four  characteristics  of  a  high-quality  and  reliable  cost 
estimate  are  that  it  is  comprehensive,  credible:  well-documented,  and  accurate. 

•  Comprehensive:  The  cost  estimate  should  include  both  government  and  contractor 
costs  of  the  program  over  its  full  life  cycle,  from  inception  of  the  program  through  design, 
development,  deployment,  and  operation  and  maintenance  to  retirement  of  the  program. 
It  should  also  completely  define  the  program,  reflect  the  current  schedule,  and  be 
technically  reasonable.  Comprehensive  cost  estimates  should  be  structured  in  sufficient 
detail  to  ensure  that  cost  elements  are  neither  omitted  nor  double  counted.  Finally, 
where  information  is  limited  and  judgments  must  be  made,  the  cost  estimate  should 
document  all  cost-influencing  ground  rules  and  assumptions. 

•  Credible:  The  cost  estimate  should  discuss  any  limitations  of  the  analysis  because  of 
uncertainty  or  biases  surrounding  data  or  assumptions.  Major  assumptions  should  be 
varied,  and  other  outcomes  recomputed  to  determine  how  sensitive  they  are  to  changes 
in  the  assumptions.  Risk  and  uncertainty  analysis  should  be  performed  to  determine  the 
level  of  risk  associated  with  the  estimate.  Further,  the  estimate’s  cost  drivers  should  be 
crosschecked,  and  an  independent  cost  estimate  conducted  by  a  group  outside  the 
acquiring  organization  should  be  developed  to  determine  whether  other  estimating 
methods  produce  similar  results. 

•  Well-documented:  A  good  cost  estimate — while  taking  the  form  of  a  single  number — is 
supported  by  detailed  documentation  that  describes  how  it  was  derived  and  how  the 
expected  funding  will  be  spent  in  order  to  achieve  a  given  objective.  Therefore,  the 
documentation  should  capture  in  writing  such  things  as  the  source  data  used,  the 
calculations  performed  and  their  results,  and  the  estimating  methodology  used  to  derive 
each  element’s  cost.  Moreover,  this  information  should  be  captured  in  such  a  way  that 
the  data  used  to  derive  the  estimate  can  be  traced  back  to,  and  verified  against  their 
sources  so  that  the  estimate  can  be  easily  replicated  and  updated.  The  documentation 
should  also  discuss  the  technical  baseline  description  and  how  the  data  were 
normalized.  Finally,  the  documentation  should  include  evidence  that  the  cost  estimate 
was  reviewed  and  accepted  by  management. 

•  Accurate:  The  cost  estimate  should  provide  for  results  that  are  unbiased,  and  it  should 
not  be  overly  conservative  or  optimistic.  An  estimate  is  accurate  when  it  is  based  on  an 
assessment  of  most  likely  costs,  adjusted  properly  for  inflation,  and  contains  few,  if  any, 
minor  mistakes.  Among  other  things,  the  estimate  should  be  grounded  in  a  historical 
record  of  cost  estimating  and  actual  experiences  on  other  comparable  programs. 

Table  9  provides  our  detailed  assessment  of  IDA’s  cost  estimating  compared  to  best  practices. 
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Table  9:  Detailed  Assessment  of  IDA’s  Cost  Estimating  Compared  to  Best  Practices 


Characteristic 

Overall 

Assessment* 

Best  Practice 

Individual 

Assessment 

Comprehensive 

Substantially 

Met 

The  cost  estimate  includes  all  life  cycle  costs 

Substantially 

Met 

The  cost  estimate  completely  defines  the  program,  reflects  the 
current  schedule,  and  is  technically  reasonable 

Partially  Met 

The  cost  estimate  is  product-oriented,  traceable  to  the 
statement  of  work/objective,  and  at  an  appropriate  level  of  detail 
to  ensure  that  cost  elements  are  neither  omitted  nor  double- 
counted. 

Substantially 

Met 

The  estimate  documents  all  cost-influencing  ground  rules  and 
assumptions. 

Partially  Met 

Well  documented 

Partially  Met 

The  documentation  should  capture  the  source  data  used,  the 
reliability  of  the  data,  and  how  the  data  were  normalized. 

Partially  Met 

The  documentation  describes  in  sufficient  detail  the  calculations 
performed  and  the  estimating  methodology  used  to  derive  each 
element’s  cost. 

Partially  Met 

The  documentation  describes  step  by  step  how  the  estimate 
was  developed  so  that  a  cost  analyst  unfamiliar  with  the 
program  could  understand  what  was  done  and  replicate  it 

Partially  Met 

The  documentation  discusses  the  technical  baseline  description 
and  the  data  in  the  baseline  is  consistent  with  the  estimate 

Partially  Met 

The  documentation  provides  evidence  that  the  cost  estimate 
was  reviewed  and  accepted  by  management. 

Substantially 

Met 

Accurate 

Partially  Met 

The  cost  estimate  results  are  unbiased,  not  overly  conservative 
or  optimistic  and  based  on  an  assessment  of  most  likely  costs. 

Partially  Met 

The  estimate  has  been  adjusted  properly  for  inflation. 

Substantially 

Met 

The  estimate  contains  few,  if  any,  minor  mistakes. 

Substantially 

Met 

The  cost  estimate  is  regularly  updated  to  reflect  significant 
changes  in  the  program  so  that  it  is  always  reflecting  current 
status. 

Not  Applicable 

Variances  between  planned  and  actual  costs  are  documented, 
explained,  and  reviewed. 

Not  Applicable 

The  estimate  is  based  on  a  historical  record  of  cost  estimating 
and  actual  experiences  from  other  comparable  programs. 

Partially  Met 

The  estimating  technique  for  each  cost  element  was  used 
appropriately. 

Partially  Met 

Credible 

Substantially 

Met 

The  cost  estimate  includes  a  sensitivity  analysis  that  identifies  a 
range  of  possible  costs  based  on  varying  major  assumptions, 
parameters,  and  data  inputs. 

Partially  Met 

A  risk  and  uncertainty  analysis  was  conducted  that  quantified 
the  imperfectly  understood  risks  and  identified  the  effects  of 
changing  key  cost  driver  assumptions  and  factors. 

Substantially 

Met 

Major  cost  elements  were  cross-checked  to  see  whether  results 
were  similar. 

Substantially 

Met 

An  independent  cost  estimate  was  conducted  by  a  group 
outside  the  acquiring  organization  to  determine  whether  other 
estimating  methods  produce  similar  results. 

Not  Applicable 

Source:  GAO  analysis  of  IDA’s  data 

*  Not  Met  -  IDA  provided  no  evidence  that  satisfies  any  of  the  criterion,  Minimally  Met  -  IDA  provided  evidence  that 
satisfies  a  small  portion  of  the  criterion,  Partially  Met  -  IDA  provided  evidence  that  satisfies  about  half  of  the  criterion, 
Substantially  Met  -  IDA  provided  evidence  that  satisfies  a  large  portion  of  the  criterion,  and  Met  -  IDA  provided 
complete  evidence  that  satisfies  the  entire  criterion. 
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Enclosure  III:  Objectives,  Scope,  and  Methodology 


To  conduct  this  work,  we  obtained  IDA’s  September  2013  cost  benefit  analysis  study  of  the 
C-130  AMP  program  and  reviewed  it  to  determine  the  data  and  assumptions  that  IDA  used  to 
reach  its  conclusions  and  recommendations.  To  analyze  IDA’s  cost  estimating,  we  relied  on 
GAO’s  Cost  Estimating  and  Assessment  Guide.  In  developing  this  guide,  our  cost  experts 
assessed  measures  consistently  applied  by  cost-estimating  organizations  throughout  the  federal 
government  and  industry  and  considered  best-practices  for  the  development  of  reliable  cost- 
estimates.  We  analyzed  the  cost  estimating  practices  used  by  the  IDA  in  its  analysis  of  the 
C-130  AMP  and  alternative  options  against  these  best  practices.  For  reporting,  we  collapsed 
these  best  practices  into  four  general  characteristics  for  sound  cost  estimating:  comprehensive, 
well  documented,  accurate,  and  credible. 

We  assigned  each  characteristic  a  rating  of  either  met,  substantially  met,  partially  met,  minimally 
met,  or  not  met.  We  determined  the  overall  assessment  rating  by  assigning  each  individual 
rating  a  number:  not  met  =  1,  minimally  met  =  2,  partially  met  =3,  substantially  met  =  4,  and  met 
=  5.  Then,  we  took  the  average  of  the  individual  assessment  ratings  to  determine  the  overall 
rating  for  each  of  the  four  characteristics.  The  resulting  average  becomes  the  Overall 
Assessment  as  follows:  not  met  =  1.0  to  1.4,  minimally  met  =  1.5  to  2.4,  partially  met  =  2.5  to 
3.4,  substantially  met  =  3.5  to  4.4,  and  met  =  4.5  to  5.0.  A  cost  estimate  is  considered  reliable  if 
the  overall  assessment  ratings  for  each  of  the  four  characteristics  are  substantially  or  fully  met. 
If  any  of  the  characteristics  are  not  met,  minimally  met,  or  partially  met,  then  the  cost  estimate 
does  not  fully  reflect  the  characteristics  of  a  high-quality  estimate  and  cannot  be  considered 
reliable.  Because  the  cost  estimates  were  developed  for  a  cost-benefit  analysis  and  not  for  a 
program  office  estimate,  we  did  not  assess  two  areas  within  the  “accurate”  characteristic — 
regular  updating  of  cost  estimates  and  tracking  cost  variances — and  one  area  within  the 
credible  characteristic — independent  cost  estimates. 

Further,  we  interviewed  agency  officials  from  IDA,  the  Air  Force  Cost  Analysis  Agency,  the 
Assistant  Secretary  of  the  Air  Force  Acquisition  Office,  and  the  Director  of  the  Office  of  the 
Secretary  of  Defense’s  Cost  Assessment  and  Program  Evaluation  office  to  discuss  the  IDA 
study  results.  In  addition,  we  reviewed  IDA’s  C-130  AMP  annex  report  to  determine  the 
methodologies  used  to  support  findings  in  the  September  2013  report. 

Given  that  IDA  reported  finding  Option  A  nearly  (if  not  as)  operationally  effective  as  AMP,  and 
that  costs  were  the  apparent  driving  factor  in  IDA’s  recommendation  that  the  Air  Force  not 
pursue  AMP,  we  limited  our  assessment  of  IDA’s  cost  estimating  to  AMP  and  Option  A. 
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Enclosure  IV:  Comments  from  the  Department  of  Defense 


ASSISTANT  SECRETARY  OF  DEFENSE 
3015  DEFENSE  PENTAGON 
WASHINGTON,  DC  20301-3015 

16  MAY  2014 


ACQUISITION 


Mr.  Michael  J.  Sullivan 

Director,  Acquisition  and  Sourcing  Management 
U.S.  Government  Accountability  Office 
441  G  Street,  N.W. 

Washington,  DC  20548 

Dear  Mr.  Sullivan: 

This  is  the  Department  of  Defense  (DoD)  response  to  the  GAO  Draft  Report, 
GAO-14-547R,  'DEFENSE  ACQUISITIONS:  Assessment  of  Institute  for  Defense  Analyses 
C-130  Avionics  Modernization  Program  (AMP)  Analysis/  dated  April  30,  2014  (GAO  Code 
121191).  The  Department  has  received  the  draft  report  and  would  like  to  include  the  following 
comments. 

The  Department  appreciates  GAO  validating  the  Institute  for  Defense  Analyses’  (IDA) 
cost  analysis  on  the  C-130  Avionics  Modernization  Program  alternatives.  When  reviewing 
IDA’s  recommendations,  DoD  took  into  consideration  the  referenced  data  sources  that  supported 
IDA’s  conclusions.  As  a  result,  the  Department  did  not  pursue  the  costly  C-130  AMP  option  and 
pursued  less  costly  alternatives. 

Sincerely, 
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Enclosure  V:  Comments  from  the  Institute  for  Defense  Analyses 


IDA  Response  to  GAO  Assessment  of 

IDA  Paper  P-5062,  C-l 30  Avionics  Modernization 

Analysis 

GAO  states  that  "IDA's  cost  analysis  generally  followed  best  practices  for  high  quality  cost  estimates, 
which  are  critical  to  the  success  of  any  program."  IDA  agrees  with  GAO's  assessment,  having  completed 
a  rigorous  analysis  of  both  cost  and  effectiveness  for  several  C-130  avionics  modernization  alternatives. 

GAO  found  that  "IDA  substantially  met  two  of  the  characteristics  expected  of  high-quality  cost  estimates 
-that  they  be  comprehensive  and  credible."  While  IDA  believes  these  two  assessments  should  have 
been  rated  "Met,"  IDA  agrees  with  GAO  that  its  independent  assessment  was  both  comprehensive  and 
credible. 

GAO  also  found  that  IDA  "partially  met  the  remaining  two  characteristics  -  being  well-documented  and 
accurate."  IDA  strongly  believes  these  two  assessments  should  have  been  rated  higher,  and  has 
provided  GAO  detailed  justification. 

Regarding  documentation,  GAO  found  that  "the  study  documentation  described  the  estimating 
methodology  used  to  derive  each  cost  estimate"...  and  "the  documentation  described  step-by-step  how 
the  estimate  was  developed,  but  it  did  not  contain  sufficiently  detailed  data  to  allow  a  cost  analyst 
unfamiliar  with  the  program  to  replicate  the  estimate."  The  IDA  report  scrupulously  identified  all  data 
sources,  but  did  not  in  many  cases  copy  the  actual  source  data  itself  into  the  report.  IDA  referenced  the 
source,  but  did  not  always  repeat  the  actual  data  in  the  IDA  report  itself,  especially  when  the  data  could 
be  easily  found  by  other  analysts.  IDA  believes  that  a  source  reference  itself  is  appropriate  for  a  report 
of  this  nature. 

Regarding  accuracy,  GAO  stated  that  "the  estimates  were  based  on  actual  costs  from  the  C-130  AMP 
and  other  programs,  program  office  estimates  for  C-130  AMP  and  other  similar  programs,  cost 
estimating  relationships,  and  factors,  but  because  the  historical  source  data  were  not  included,  we  could 
not  assess  the  reliability  and  relevance  of  the  data."  IDA  strongly  believes  they  produced  accurate  cost 
estimates  using  sound  methodology  as  described  in  detail  in  their  report.  GAO  did  not  in  fact  find  any 
evidence  that  the  IDA  assessment  was  not  accurate.  GAO's  primary  reason  for  its  "Partially  Met"  rating 
was  that  they  could  not  verify  the  accuracy  of  the  analysis  because  all  source  documentation  was  not 
included  in  the  IDA  report. 

The  IDA  independent  cost-benefit  assessment  determined  there  are  lower  cost  options  of  nearly  as 
much  capability  as  C-130  AMP.  The  IDA  report  thus  recommended  that  the  US  Air  Force  pursue  a 
reduced  scope  option,  such  as  Options  A  or  B  and  not  pursue  C-130  AMP.  Nothing  in  the  GAO 
assessment  undermines  these  findings  and  recommendations. 


(121191) 
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This  is  a  work  of  the  U.S.  government  and  is  not  subject  to  copyright  protection  in  the 
United  States.  The  published  product  may  be  reproduced  and  distributed  in  its  entirety 
without  further  permission  from  GAO.  However,  because  this  work  may  contain 
copyrighted  images  or  other  material,  permission  from  the  copyright  holder  may  be 
necessary  if  you  wish  to  reproduce  this  material  separately. 
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